Control method of mobile terminal apparatus

ABSTRACT

A mobile device system that includes left and right earphones, and a mobile device connectable to the left and right earphones. Each of the left and right earphones includes at least one of an acceleration sensor and a geomagnetic sensor. The mobile device includes a controller that monitors a wearing state of the individual left and right earphones by a user on a basis of output from the least one of the acceleration sensors and the geomagnetic sensors, and controls a behavior of the mobile device connected to the left and right earphones according to the monitored wearing state of the left and right earphones by the user.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of and claims the benefit of priorityunder 35 U.S.C. §120 from U.S. application Ser. No. 14/035,605, filedSep. 24, 2013, which is related to and claims priority under 35 U.S.C.§119(e) to Provisional Application Ser. No. 61/705,860, filed Sep. 26,2012, the entire contents of each of which is hereby incorporated hereinby reference.

BACKGROUND

1. Field

The present specification relates to monitoring the earphone wearingstate of a user wearing headphones that include left and rightearphones, as well as to a method of controlling a mobile deviceaccording to the wearing state.

2. Description of the Related Art

Typically, headphones are used as an apparatus for the purpose of a userconverting an audio signal output from an audio playback apparatus intoa sound wave (audible sound), generally to listen to music or other suchaudio alone. The headphones in this specification are connected to suchan audio playback apparatus in a wired or wireless manner, and includestereo types provided with a pair of left and right earphones. Anearphone herein refers to the component of headphones worn so as tobring a speaker close to one of the user's ears.

Hitherto, technology providing audio-based navigation to pedestrianswearing headphones has been proposed (see PTL 1). With this technology,the angle of cranial rotation with respect to the direction in which auser is traveling (the front-to-back direction of the user's body) iscomputed as follows. Namely, established laser range-finding methods areused to detect the shortest distance from the user's left shoulder tothe left side of the headphones, and also to detect the shortestdistance from the user's right shoulder to the right side of theheadphones. Additionally, a sensor worn near the base of the head isused to detect the rotational direction of the head (right-handedturning or left-handed turning as viewed from above). The angle ofcranial rotation with respect to the user's travel direction is computedon the basis of these two shortest distances and cranial rotation thusdetected. The position of the sound source is corrected on the basis ofthe angle of cranial rotation.

CITATION LIST

[Patent Literature][PTL 1] Japanese Unexamined Patent ApplicationPublication No. 2002-5675

Meanwhile, technology for headphones including left and right earphonesthat checks whether or not each earphone is being worn has not existedhitherto. If a mobile device or other device with connected headphonescould ascertain the wearing state of each earphone, the possibility ofdiversifying mobile device control could be anticipated.

Given such background, the inventor has recognized the need for a newmethod of controlling a mobile device related to the wearing state ofleft and right earphones.

BRIEF SUMMARY

According to an exemplary embodiment, there is provided a method ofcontrolling a mobile device that includes a step of monitoring thewearing state of left and right earphones by a user, and a step ofcontrolling the behavior of a mobile device connected to the left andright earphones according to the wearing state of the left and rightearphones by the user.

The wearing states of the left and right earphones may include a firststate in which both earphones are being worn, a second state in whichonly the left or right earphone is being worn, and a third state inwhich both the left and right earphones are removed, for example.

In the step of controlling the behavior of the mobile device, at leastone of launching or shutting down an application, switching theoperational mode in an application, and switching the display in anapplication may be conducted, according to a change in the wearing stateof the left and right earphones.

In the step of monitoring the wearing state of the left and rightearphones by the user, the wearing state of the left and right earphonesmay be detected on the basis of output from left and right accelerationsensors respectively provided in the left and right earphones, and/oroutput from left and right geomagnetic sensors respectively provided inthe left and right earphones.

The method of controlling a mobile device may further include a step ofcausing the user to execute a nodding gesture, in which the user looksdirectly ahead with respect to his or her body, rotates his or her headforward from an upright state by a given angle or more, and then returnshis or her head to its original upright state, and a step of checkingwhether each earphone is being worn, on the basis of output during thenodding gesture from left and right acceleration sensors respectivelyprovided in the left and right earphones.

Also, a mobile device according to an exemplary embodiment includes leftand right earphones, and a mobile device connectable to the left andright earphones. Each of the left and right earphones includes anacceleration sensor and/or a geomagnetic sensor. The mobile deviceincludes a controller that detects the wearing state of the individualleft and right earphones by the user on the basis of output from theacceleration sensors and/or the geomagnetic sensors, and controls thebehavior of the mobile device connected to the left and right earphonesaccording to the wearing state of the left and right earphones.

According to an exemplary embodiment, it is possible to detect the stateof whether left and right earphones are being worn by the user, andcontrol a mobile device according to the detected wearing state.

BRIEF DESCRIPTION OF THE DRAWINGS

FIGS. 1A and 1B and 1C are diagrams illustrating a diagrammaticconfiguration of a mobile device equipped with wired and wireless stereoheadphones in the exemplary embodiment, respectively.

FIGS. 2A and 2B are diagrams illustrating exemplary exteriors of varioustypes of stereo headphones.

FIGS. 3A and 3B are diagrams illustrating states of a user wearing aheadphone according to the exemplary embodiment.

FIG. 4 is a diagram for explaining the respective action of ageomagnetic sensor and an acceleration sensor built into (the housingof) an earphone.

FIGS. 5A and 5B are diagrams for explaining relationships of variousvectors and various angles in a three-dimensional coordinate system inwhich an earphone is disposed.

FIGS. 6A and 6B are another set of diagrams for explaining relationshipsof various vectors and various angles in a three-dimensional coordinatesystem in which an earphone is disposed.

FIGS. 7A and 7B are diagrams for explaining acceleration sensor actionother than detecting a gravity vector.

FIGS. 8A, 8B, and 8C are diagrams for explaining an example of jointlyusing a gyroscope as a sensor.

FIG. 9 is a block diagram illustrating an exemplary configuration of amobile device in an exemplary embodiment.

FIG. 10 is a diagram illustrating an exemplary configuration of a mobiledevice that uses wireless left and right earphones.

FIG. 11 is a graph illustrating change in the gravity-inducedacceleration components Gys and Gxs for left and right accelerationsensors during a nodding gesture in which a user is correctly wearingleft and right earphones.

FIG. 12 is a graph illustrating asymmetry in the output from left andright acceleration sensors.

FIG. 13 is a graph illustrating the Xs axis output Gxs (or the Ys axisoutput Gys) from an acceleration sensor during a nodding gesture in thecase where an earphone is being worn on the correct side.

FIG. 14 is a diagram for explaining the case of jointly using agyroscope with an acceleration sensor.

FIG. 15 is a flowchart illustrating exemplary behavior of a mobiledevice in an exemplary embodiment.

FIG. 16 is a flowchart illustrating an exemplary modification of theprocess illustrated in FIG. 15.

FIGS. 17A, 17B, 17C, and 17D are diagrams for explaining exemplarycontrol of a mobile device according to the wearing state of left andright earphones.

FIGS. 18A and 18B are diagrams illustrating exemplary displays that varyaccording to the wearing state of left and right earphones, which may beimplemented together with, or independently of, the exemplary controlfor audio navigation illustrated in FIGS. 17A to 17D.

FIG. 19 is a diagram illustrating a state where an earphone is worn on auser's head.

FIG. 20 is a diagram re-illustrating a state in which a user is wearingan earphone, as well as a sensor coordinate system and user coordinatesystem in such a state.

FIG. 21 is a diagram for explaining axis transformation by rotationabout the Z axis of an earphone.

FIG. 22 is a diagram for explaining axis transformation by rotationabout the X axis of an earphone.

FIG. 23 is a diagram for explaining axis transformation by rotationabout the Y axis of an earphone.

FIG. 24 is a diagram for explaining a nodding gesture that a user ismade to execute in a state of wearing an earphone.

FIG. 25 is a diagram illustrating a waveform of the output from agyroscope during a nodding gesture.

FIG. 26 is a diagram illustrating an exemplary configuration in whichthe functionality of a mobile device is built into headphones.

DETAILED DESCRIPTION

Hereinafter, an exemplary embodiment will be described in detail andwith reference to the drawings.

In the exemplary embodiment, the wearing state of the left and rightearphones in a set of stereo headphones is monitored, and the behaviorof a mobile device connected to the left and right earphones iscontrolled according to the wearing state of the left and rightearphones. In order to do so, the individual left and right earphonesare equipped with sensors, and the wearing state of each earphone isdetected on the basis of periodically detected sensor output.

Hereinafter, a configuration for realizing the exemplary embodiment willbe described, and after that, its behavior will be described.

FIGS. 1(a), 1(b) and 1(c) illustrate a diagrammatic configuration ofmobile devices 100 a and 100 b equipped with wired and wireless stereoheadphones in the exemplary embodiment, respectively.

The wired earphones 10 aL and 10R are connected to the correspondingmobile device 100 a via a cable 18. The left and right earphones 10 bLand 10 bR are wirelessly connected to the mobile device 100 b via awireless interface using their antenna 19 and a corresponding antenna109 in the mobile device 100 b. A single antenna 19 may be shared as inFIG. 1(b) in the case where the earphones 10 bL and 10 bR are joined bya cable or other means as illustrated in FIGS. 2(a) (b) discussed later.In the case where the left and right earphones 10 cL and 10 cR areseparated (independent) from each other as illustrated in FIG. 1(c),both earphones are separately equipped with antennas 19L and 19R (andcommunication circuits).

FIGS. 2(a) and 2(b) illustrate exemplary exteriors of various types ofstereo headphones.

FIGS. 2(a) and 2(b) respectively illustrate headphones (ear receivers)10 a and 10 b which may be referred to as inner-ear or canal headphones,and which include ear canal plugs 17 aL, 17 aR, 17 bL, and 17 bR worninside the user's ear canal.

The wired headphones 10 a illustrated in FIG. 2(a) include respectivehousings 15 aL and 15 aR, ear canal plugs 17 aL and 17 aR projectingfrom their sides, and left and right earphones 10 aL and 10 aR thatinclude a cable 18 leading out from the bottom of their respectivehousings. Sensor devices 16 aL and 16 aR are housed inside the housings15 aL and 15 aR of the earphones 10 aL and 10 aR. In the exemplaryembodiment, each sensor device at least includes a geomagnetic sensor 11and an acceleration sensor 12.

The wireless headphones 10 b illustrated in FIG. 2(b) include respectivehousings 15 bL and 15 bR, ear canal plugs 17 bL and 17 bR projectingfrom their sides, and left and right earphones 10 bL and 10 bR thatinclude a cable 18 i connected between their respective housings 15 bLand 15 bR. Sensor devices 16 bL and 16 bR are housed inside the housings15 bL and 15 bR of the earphones 10 bL and 10 bR. In the exemplaryembodiment, each sensor device at least includes a geomagnetic sensor11, an acceleration sensor 12, and a wireless communication unit(discussed later). The cable 18 i is unnecessary in the case where boththe left and right earphones 10 bL and 10 bR each include a wirelesscommunication unit independently (this corresponds to FIG. 1(c)).

Hereinafter, the exemplary embodiment will be described takingheadphones of the type illustrated in FIGS. 2(a) and 2(b) as an example,but the following similarly applies to other types of headphones.

FIGS. 3(a) and 3(b) illustrate states of a user wearing headphonesaccording to the exemplary embodiment. This example corresponds to thestate of wearing a pair of earphones on the left and right ears.Hereinafter, the left and right earphones 10L and 10R will be simplydesignated the earphone 10 when not particularly distinguishing them.

An arbitrary application that generates sound or voice suing earphonesis relevant as an example of an application applying the exemplaryembodiment. Applications such as audio navigation, games, and musicplayers are a possibility.

Potential wearing states of left and right earphones include a firststate in which both earphones are being worn, a second state in whichonly the left or right earphone is being worn, and a third state inwhich both the left and right earphones are removed.

Controlling the behavior of a mobile device involves conducting at leastone of launching or shutting down an application, switching theoperational mode in an application, and switching the display in anapplication, according to a change in the wearing state of the left andright earphones.

When monitoring the wearing state of left and right earphones by theuser, the wearing state of the left and right earphones is detected onthe basis of output from the left and right acceleration sensorsrespectively on board the left and right earphones, and/or output fromthe left and right geomagnetic sensors respectively on board the leftand right earphones.

However, depending on the application, in some cases there may be anattempt to accurately compute the direction in which the user's face isfacing (the orientation F of the face) while the earphones 10 are beingworn on the user's head.

Even in a state of being worn on the user's head, the earphones 10 mayrotate within an angular range to some extent, mostly about an axisgiven by a line joining the left and right ears. FIGS. 3(a) and 3(b)illustrate states where an earphone 10 is worn on the user's head atdifferent rotational angles. As illustrated, whereas the orientation Fof the user's face and the forward direction specific to the earphone 10(the forward vector Vf) may match in some cases, in other cases they maynot match.

For an earphone 10 worn on the user's head as illustrated in FIGS. 3(a)and 3(b), the direction in which the user's face is facing (theorientation F of the face) may be determined as follows. Specifically,the forward vector Vf of the earphone 10 nearly matches the facialorientation F in the case where the user is wearing the earphone 10 suchthat its lengthwise direction is aligned with a direction nearlyvertical from the ground (the vertical direction), as illustrated inFIG. 3(a). Meanwhile, the actual orientation F of the user's face maystill be computed by correcting the forward vector Vf of the earphone 10on the basis of the sensor output from the acceleration sensor 12, evenin the case where a tilt (wearing angle error) is produced in theearphone 10 due to how the earphone 10 is attached to the head, asillustrated in FIG. 3(b). Herein, although the rotation of the earphoneabout an axis given by the direction joining the user's ears is taken tobe the problem, an earphone may also potentially rotate in thehorizontal plane about an axis given by the vertical direction. Thislatter rotation in particular affects detection of the orientation ofthe user's face.

As discussed later, the 3-axis acceleration sensor 12 and the 3-axisgeomagnetic sensor 11 on board an earphone 10 in the exemplaryembodiment may also be used as an orientation detecting unit fordetecting information such as the current state of the user's head,specifically the orientation F of the user's face, or in other words thedirection (heading) in which the front of the head (the face) is facing.

FIG. 4 is a diagram for explaining the respective action of thegeomagnetic sensor 11 and the acceleration sensor 12 built into (thehousing 15 of) an earphone 10.

The 3-axis geomagnetic sensor 11 ascertains the direction ofgeomagnetism, or in other words a geomagnetic vector Vt, given thecurrent orientation of (the housing 15 of) the earphone 10 housing the3-axis geomagnetic sensor 11.

Herein, for the sake of explanation, take an Xs axis, a Ys axis, and aZs axis to be three mutually orthogonal axes in a localthree-dimensional coordinate system specific to the earphone 10 (inother words, specific to the sensor; a sensor coordinate system). The Xsaxis corresponds to the front and back direction of the earphone, whilethe Ys axis corresponds to the top and bottom direction of the earphone.The Zs axis is the axis orthogonal to the Xs axis and the Ys axis. TheZs axis mostly corresponds to the direction along the line joining theuser's ears when the user wears the earphone 10. In the case where theearphone 10 is an earphone 10L worn on the user's left ear, anear-contacting portion (ear canal plug) is disposed on the side of thehousing 15 in the negative direction of the Zs axis. Conversely, in thecase of an earphone 10R worn on the user's right ear, an ear-contactingportion is disposed on the side of the housing 15 in the positivedirection of the Zs axis. The Xs axis is orthogonal to both the Ys axisand the Zs axis. In this example, the positive direction of the Xs axisis taken to match the forward vector Vf of the earphone 10. Ageomagnetic vector Vt typically may be decomposed into Xs, Ys, and Zsaxis components as illustrated.

The 3-axis acceleration sensor 12 ascertains the direction of gravity,or in other words a gravity vector G, given the current orientation of(the housing 15 of) the earphone 10 housing the 3-axis accelerationsensor 12 in a still state. The gravity vector G matches the downwardvertical direction. The gravity vector G likewise may be decomposed intoXs, Ys, and Zs axis components as illustrated.

By using the 3-axis acceleration sensor 12 in this way, it is possibleto detect the orientation of the earphone 10 in the three-dimensionalspace in which (the housing 15 of) the earphone 10 is disposed. Bycomparing the sensor output and its variation according to theorientations of the left and right earphones, it is possible to detectthe wearing state of left and right earphones 10. In addition, it ispossible to detect the wearing state of left and right earphones 10 byalso using left and right 3-axis geomagnetic sensors 11.

First, in the state in which the left and right earphones are both beingworn (the first state), it is conceivable that the output from theacceleration sensors 12 (or the absolute values thereof) will mostlymatch, and their variation will also indicate common tendencies.Likewise for the left and right geomagnetic sensors 11, it isconceivable that the detected geomagnetic directions will mostly match,and their variation will also indicate common tendencies. Consequently,it may be determined that both earphones are in a worn state in the casewhere, after both earphones are worn initially, variation in the outputfrom the acceleration sensors 12 and/or the geomagnetic sensors 11 aresimilar for left and right.

Next, in the case where either the left or right earphone is removedstarting from the above state, it is conceivable that variation in theacceleration sensor 12 and the geomagnetic sensor 11 will be small forthe earphone that was not removed, whereas variation in the accelerationsensor 12 and the geomagnetic sensor 11 will be extremely large bycomparison for the earphone that was removed. Consequently, when such arelationship is detected between the left and right output from theacceleration sensors 12 and the geomagnetic sensors 11, it may bedetermined that the headphones have transitioned from the first state tothe second state, in which only either the left or right earphone isbeing worn.

In the case where only the output from the acceleration sensor 12 andthe geomagnetic sensor 11 of the earphone being worn changes greatlystarting from such a second state, it may be determined that theheadphones have transitioned to the third state, in which both the leftand right earphones have been removed.

Furthermore, according to an orientation detecting unit of the exemplaryembodiment, it is possible to detect the heading (such as north, south,east, or west) in which the front of (the housing 15 of) an earphone 10is facing. It is possible to detect the wearing state of the left andright earphones by also detecting the heading in which the fronts of theleft and right earphones are facing and its variation. However, in theexemplary embodiment, it is not strictly necessary to actually computethe heading.

Note that the user may also be made to perform a nodding gesture asdiscussed later, in order to initially check the state in which the useris wearing the headphones. In this specification, a nodding gesturerefers to a gesture in which the user looks directly ahead with respectto his or her body, rotates his or her head forward from an uprightstate by a given angle or more, and then returns his or her head to itsoriginal upright state. With such a gesture, it is possible to check, onthe basis of the sensor output during the nodding gesture, whether ornot the user is wearing each earphone, and also whether the earphonesare being correctly worn on the left and right.

FIGS. 5(a) and 5(b) are diagrams for explaining relationships of variousvectors and various angles in a three-dimensional coordinate system inwhich an earphone is disposed.

As illustrated in FIG. 5(a), take an Xu axis, a Yu axis, and a Zu axisto be the mutually orthogonal axes of a coordinate system for athree-dimensional space in which an earphone 10 is disposed, or in otherwords, the three-dimensional space in which the user is positioned. Thiscoordinate system is called the user coordinate system (Xu, Yu, Zu) todistinguish it from the sensor coordinate system (Xs, Ys, Zs) as above.The variables used in both these coordinate systems will bedistinguished with the subscripts s (sensor) and u (user). The Xu axiscorresponds to the front and back direction of the user, while the Yuaxis corresponds to the top and bottom direction of the user. The Zuaxis is the axis orthogonal to the Xu axis and the Yu axis. The negativedirection of the Yu axis lies along the gravity vector G. The planeorthogonal to the gravity vector G is the XuZu plane, and corresponds toa horizontal plane 31 in the space where the user is positioned. For thesake of convenience, the Zu axis is taken to match the Zs axis.

As discussed earlier, when the user wears the earphone 10, the top andbottom direction (lengthwise direction) of the earphone 10 does notnecessarily match the vertical direction. Likewise, the example in FIG.5(a) illustrates an example where the vertical direction (the directionalong the Yu axis) and the Ys axis direction of the sensor coordinatesystem do not match.

For the sake of convenience, imagine a plane 33 containing a face of thehousing 15 of an earphone 10 (the face that comes into contact with theuser's ear), as illustrated in FIG. 5(a). The direction of the linewhere the plane 33 and the horizontal plane 31 intersect (the vectorVfxz) may be determined to be the orientation F of the user's face. Theorientation F of the face computed in this way may include some degreeof error with respect to the exact orientation of the face, due to howthe earphone is worn. However, this error is considered to be within anacceptable range for many applications.

The user's nodding gesture discussed above may be used as a method ofmore accurately computing the orientation F of the face. In other words,it may be configured such that when the user wears headphones, the useris requested to perform a nodding gesture with his or her head in theforward direction, and the error between the forward direction of theheadphones and the orientation of the user's face is computed on thebasis of output from the acceleration sensor in a state before thenodding and a state at the maximum nodding angle. In this case, it ispossible to accurately detect the current orientation of the face of auser wearing an earphone, and use this orientation for various controlsin relevant applications. Accurately detecting the orientation of auser's face may be conducted by detecting the wearing state and wearingangle of the earphone. Particularly, by detecting the offset anglebetween the orientation of the user's face on a horizontal plane (theforward direction) and the forward direction of the sensor mounted onboard the earphone (a specific axis), it is possible to correct theforward direction determined by the sensor. With such a method, it ispossible to detect the orientation of the user's face may be detectedwith higher precision. This specific method will be later discussed indetail.

A reference azimuth vector Vtxz is obtained from the geomagnetic vectorVt by projecting this vector onto the horizontal plane 31. The vectorVfxz on the horizontal plane 31 is specified as the vector in thedirection of an angle θf based on the reference azimuth vector Vtxz.

By using the geomagnetic sensor 11 and the acceleration sensor 12 incombination, it is possible to obtain information on the direction(heading) in which the user (the user's face) is facing which isrequired for navigation or other applications, even when the user is ina stationary state, or in other words even if the user is not moving.Also, sensors of comparatively small size may be used for these sensorswith current device technology, and thus it is possible to install suchsensors on board an earphone without difficulty.

FIGS. 6(a) and 6(b) are another set of diagrams for explainingrelationships of various vectors and various angles in athree-dimensional coordinate system in which an earphone is disposed.

Instead of computing the orientation F of the face as described withFIG. 5(a), the forward vector Vf along the X axis direction may also beapproximately set, as illustrated in FIG. 6(a). In this example, theforward vector Vf matches the positive direction of the Xs axis. Themagnitude of the forward vector Vf is arbitrary (or a unit vector). Thedirection indicated by a vector Vfxz obtained by projecting the forwardvector Vf onto the horizontal plane, or in other words the XuZu plane31, may be determined to be the orientation F of the user's face. Theorientation F of the face computed according to the forward vector Vfdoes not necessarily match the orientation F of the face described withFIG. 5(a), and likewise may include error with respect to the exactorientation of the face. However, the orientation F of the face may becomputed quickly and easily.

In either case, if the user moves his or her head, an earphone 10 beingworn on the head moves together with it. In response to such movement ofthe head, the current vertical direction with respect to the earphone 10(the gravity vector G) is detected at individual points in time. Also,as the head moves, the plane 33 (or the forward vector Vf) in the usercoordinate system changes, and a new corresponding vector Vfxz (ororientation F of the face) is determined.

FIGS. 7(a) and 7(b) are diagrams for explaining action of theacceleration sensor 12 besides detecting a gravity vector.

As illustrated in FIG. 7(a), besides detecting constant accelerationssuch as gravity, the acceleration sensor 12 is also able to detectdynamic accelerations that accompany movement. For example, in the casewhere an object moves, positive acceleration is imparted to that objectfrom a stationary state, and negative acceleration is imparted when theobject stops. For this reason, the acceleration of an object isdetected, and from the integral thereof it is possible to compute themovement velocity and the movement distance, as illustrated in FIG.7(b). However, since the acceleration does not change in the case ofuniform motion, the movement state cannot be detected unless anacceleration from a stationary state is detected. Also, due to theconfiguration of the acceleration sensor 12, rotations cannot bedetected in the case of rotation about the gravity vector as axis.

In contrast, FIGS. 8(a), 8(b), and 8(c) will be used to explain anexample of jointly using a gyroscope as a sensor.

As illustrated in FIG. 8(a), the gyroscope 13 is a sensor that detectsangular velocity about the three axes Xs, Zs, and Ys (roll, pitch, andyaw), and is able to detect the rotation of an object. In addition, thegeomagnetic sensor 11 is able to ascertain the heading in which theobject faces, on the basis of a geomagnetic vector as discussed earlier.However, in cases where the magnetic field lines are not in a constantdirection, such as when near a magnetized steel frame, it may becomeimpossible to recognize the correct heading in some cases when theobject rotates while moving. For this reason, the rotational state maybe detected with the gyroscope only in cases of movement like thatillustrated in FIG. 8(c). Herein, the object is represented by a compassneedle for the sake of convenience.

Consequently, by jointly using a gyroscope 13 together with the aboveacceleration sensor 12 and geomagnetic sensor 12 as sensors installed onboard an earphone 10, it may be configured to supplement the output fromboth sensors.

In this way, although it is possible to detect the orientation F of theuser's face in real-time and with some degree of precision using only ageomagnetic sensor and an acceleration sensor 12, by jointly using agyroscope (gyro sensor) it becomes easy to track even comparatively fastchanges in direction by the user.

Hereinafter, a configuration of a mobile device and headphones(earphones) shared by both of the exemplary embodiments will bedescribed. A variety of apparatus are known as mobile devices, such asmobile phone handsets, audio players, video players, television sets,radio receivers, electronic dictionaries, and game consoles.

FIG. 9 is a block diagram illustrating an exemplary configuration of amobile device 100 a in the exemplary embodiment. The mobile device 100 ais equipped with wired stereo earphones 10 aL and 10 aR. Headphonesprovided with earphones with an attached microphone is typically calleda headset. Although a microphone was not particularly illustrated in theblock diagrams or exterior views of the various earphones discussedearlier, a microphone may be built in. Although a microphone may behoused inside the housing 15, it is also possible to dispose amicrophone projecting outward therefrom or partway along the cable 18.

The mobile device 100 a includes a control line 150 and a data line 160,and is configured by various functional units like the following, whichare connected to these lines.

The controller 101 is composed of a processor made up of a centralprocessing unit (CPU) or the like. The controller 101 executes variouscontrol programs and application programs, and also conducts variousdata processing associated therewith. In the data processing, thecontroller 101 exerts communication control, audio processing control,image processing control, various other types of signal processing, andcontrol over respective units, for example.

The communication circuit 102 is a circuit for wireless communicationused when the mobile device 100 communicates with a wireless basestation on a mobile phone network, for example. The antenna 103 is awireless communication antenna used when the mobile device 100 awirelessly communicates with a wireless base station.

The display unit 104 is a component that administers a display interfacefor the mobile device, and is composed of a display device such as aliquid crystal display (LCD) or an organic electroluminescent (OEL)display. The display unit 104 may be additionally equipped with a lightemitter such as a light-emitting diode (LED).

The operable unit 105 is a component that administers an input interfaceto the user, and includes multiple operable keys and/or a touch panel.

The memory 106 is an internal storage apparatus composed of RAM andflash memory, for example. The flash memory is non-volatile memory, andis used in order to store information such as operating system (OS)programs and control programs by which the controller 101 controlsrespective units, various application programs, and compressedmusic/motion image/still image data content, as well as varioussettings, font data, dictionary data, model name information, and deviceidentification information, for example. In addition, other informationmay be stored, such as an address book registering the phone numbers,email addresses, home addresses, names, and facial photos of users, sentand received emails, and a scheduler registering a schedule for the userof the mobile device. The RAM stores temporary data as a work area whenthe controller 101 conducts various data processing and computations.

The external connection terminal 107 is a connector that connects to thecable 18 leading to the earphone 10 a.

The external apparatus connection unit 170 is a component that controlsthe reading and writing of a removable external storage apparatus 171with respect to the mobile device 100 a. The external storage apparatus171 is an external memory card such as what is called a Secure Digital(SD) card, for example. In this case, the external apparatus connectionunit 170 includes a slot into which an external memory card may beinserted or removed, and conducts reading/writing control of data withrespect to the external memory card, as well as signal processing.

The music data controller 173 is a component that reads and plays backmusic data stored in the external storage apparatus 171 or the memory106. The music data controller 173 may also be configured to be able towrite music data. Played-back music data may be converted into sound atthe earphones 10 a (10 aL and 10 aR) to enable listening.

The imaging controller 174 controls imaging by a built-in camera unit175.

The GPS controller 176 functions as a position detector for receivingsignals from given satellites with a GPS antenna 177 and obtainingposition information (at least latitude and longitude information) forthe current location.

The speaker 110 is an electroacoustic transducer for outputting receivedtelephony audio that converts an electrical signal into sound. Themicrophone unit (mic) 122 is a device for outputting telephonytransmitter audio, and converts sound into an electrical signal.

In the case where the earphones 10 a are connected to the mobile device100 a, an external speaker 421 and an external mic 422 inside theearphones 10 a are used instead of the speaker 110 and the mic 122 builtinto the device. The external speaker 421 of the earphones 10 a isconnected to an earphone terminal 121 via the cable 18.

A geomagnetic sensor 131, an acceleration sensor 132, and a gyroscope133 are also built into the mobile device 100 a. These sensors are fordetecting information such as the orientation and movement velocity ofthe mobile device 100, and are not directly used in the exemplaryembodiments.

The earphones 10 aL and 10 aR each include the external speaker 421, theexternal mic 422, an external geomagnetic sensor 411, an externalacceleration sensor 412, an external gyroscope 413, and an externalconnection controller 401. However, the external mic 422 and theexternal gyroscope 413 are not required elements in the exemplaryembodiments.

The external connection controller 401 is connected to the respectivesensors by a control line and a data line, while also being connected tothe external connection terminal 107 of the mobile device 100 via thecable 18. Preferably, output from each sensor is acquired periodicallyor as necessary in response to a request from the mobile device 100, andtransmitted to the mobile device 100 as sensor detection signals. Morespecifically, the external connection controller 401 includes variousexternal connectors such as a connector according to the standard knownas USB 2.0 (Universal Serial Bus 2.0), for example. For this reason, themobile device is also equipped with a USB 2.0 controller.

Note that the mobile device 100 a may also include various componentswhich are not illustrated in FIG. 9, but which are provided in existingmobile devices.

Regarding duplicate sensors on board the left and right earphones, thequestion of whether to use the same sensors in both the left and rightearphones or only those on one side may differ by application. Forexample, both the left and right sensors are used in order to check thewearing state of the left and right earphones in the exemplaryembodiments. It is sufficient to use only the sensors on one side in aworn state in order to detect the correct orientation of the user'sface.

FIG. 10 illustrates an exemplary configuration of a mobile device 100 bthat uses wireless left and right earphones 10 bL and 10 bR. Since theconfiguration is generally the same as that of the mobile device 100 aillustrated in FIG. 9, similar elements are denoted with the samereference signs, and duplicate description thereof will be reduced oromitted. The earphone 10 bL is equipped with an external wirelesscommunication unit 430 and an external communication antenna 431, andwirelessly communicates with the antenna 109 of a wireless communicationunit 108 in the mobile device 100 b. The wireless communication isshort-range wireless communication, and wireless communication isconducted over a comparatively short range according to a short-rangewireless communication format such as Bluetooth®, for example. Similarlyto the earphone 10 bL, the other earphone 10 bR is equipped with anexternal wireless communication unit 430 and an external communicationantenna 431, and wirelessly communicates with the antenna 109 of thewireless communication unit 108 in the mobile device 100 b. In the casewhere the earphone 10 bR and the earphone 10 bL are connected by a cable(18 i), it is sufficient to provide the external wireless communicationunit 430 and the external communication antenna 431 in only one of theearphones.

Next, the nodding gesture that the user is made to execute in theexemplary embodiment will be described. Although the nodding gesture isnot strictly required in the exemplary embodiment, causing the user toperform a nodding gesture makes it possible for the controller 101 ofthe mobile device to check, on the basis of the sensor output during thenodding gesture, whether or not the user is wearing each earphone, andalso whether the earphones are being correctly worn on the left andright.

The user is made to execute the nodding gesture as an initial gesturewhen the user puts on the earphones (headphones) and starts execution ofthe application to be used, or at a given time, such as when connectingearphones to a mobile device. For this reason, it may be configured suchthat explicit instructions for performing the nodding gesture areindicated by the user interface with a display or sound (or voice) atevery instance of such a given time. Alternatively, the user may beinformed of the necessity of a nodding gesture manually or otherwise asdetermined by the application.

FIG. 11 illustrates variation in the gravity-induced accelerationcomponents Gys and Gxs for left and right acceleration sensors during anodding gesture in which a user is correctly wearing left and rightearphones. Both graphs are obtained by monitoring the X axis and Y axissensor output from the acceleration sensors over a given interval at agiven sampling period. As the drawing demonstrates, the left and rightsensor output Gys and Gxs have waveforms of nearly the same shape. Inother words, on both the left and the right, the X axis output and the Yaxis output from the 3-axis acceleration sensor exhibits convex changeas they vary from the start time to the end time of a nodding gesture,increasing at first but then decreasing after reaching maximum valuesGys(α) and Gxs(α), and returning to their initial values. Note thatalthough the acceleration sensor output is designed to exhibit convexvariation during a nodding gesture in which the user is correctlywearing left and right earphones, the acceleration sensor output mayalso be designed to conversely exhibit concave variation in such cases.The important point is that since the output from the left and rightacceleration sensors vary similarly (symmetrically), it may bedetermined that the left and right earphones are both being worn.

Assuming that the user is stationary during the nodding gesture, it isanticipated that there will be little variation in the output from theacceleration sensor in an earphone that is not being worn. Moreover,even if the user does move, it is conceivable that the output from theacceleration sensor in an earphone being worn will differ greatly (beasymmetric to) the output from the acceleration sensor in an earphonenot being worn.

For example, in the case where only the left earphone is being worn andthe right earphone is not being worn, the output from the left and rightacceleration sensors becomes asymmetric, as illustrated in FIG. 12. Inthis example, only the output from the left acceleration sensor exhibitsconvex variation, while the output from the right acceleration sensorstays flat. Although not illustrated, in the case where only the rightearphone is being worn and the left earphone is not being worn, thegraphs in FIG. 12 are swapped left and right. Also, although the outputfrom the left and right acceleration sensors is symmetric in the casewhere a nodding gesture is performed in a state where neither of theleft and right earphones are being worn, their variation does notexhibit convex variation.

Note that ordinarily, the two earphones in a set of stereo headphonesare statically determined in advance to be a left earphone and a rightearphone, respectively. For this reason, when using the headphones, theuser puts on the headphones by visually checking the left and rightearphones. If the user mistakenly wears the headphones backwards, notonly will the left and right stereo audio be reversed, but the detectionresults based on sensor output will be off by approximately 180°, andthere is a risk of no longer being able to expect correct operation.

In the exemplary embodiment, the Xs axis output Gxs (or the Ys axisoutput Gys) from an acceleration sensor during a nodding gesture becomesconvex in the case where an earphone is being worn on the correct side,as illustrated in FIG. 13. In contrast, that output becomes concave inthe case of being worn on the incorrect side. Consequently, byrecognizing the shape from the acceleration sensor, it is possible todetermine whether each earphone is being worn on the correct side.

In this way, it is possible to determine whether or not an earphone isbeing worn on the correct side, according to whether the sensor outputfor a specific axis (herein, the Xs axis or the Ys axis) of anacceleration sensor exhibits convex variation or concave variationduring a nodding gesture.

FIG. 14 is a diagram for explaining the case of jointly using agyroscope with an acceleration sensor. The motion of a gyroscope aboutthe axis of the nodding rotational direction is reversed when eachearphone is worn on the correct side and worn on the incorrect side. Inother words, the phase of the waveform in the gyroscope output differsby 180° when an earphone is worn on the left and worn on the right. Inthe example in the drawing, it is possible to determine whether anearphone is being worn correctly or incorrectly depending on whether thewaveform changes from negative to positive or from positive to negativeat the zero-crossing.

Next, FIG. 15 illustrates exemplary behavior of a mobile device in theexemplary embodiment. This process is executed due to the controller 101of the mobile device executing a computer program stored in the memory106 or elsewhere.

First, a given initialization process for the left and right earphonesis executed (S11). This initialization process includes the noddinggesture by the user discussed earlier, and at least monitors the outputfrom the acceleration sensors of the left and right earphones during thenodding gesture. By causing the user to perform a nodding gesture whilewearing the earphones, it is checked whether each earphone is being wornon the correct side. In the case where an earphone is a predeterminedleft-ear or right-ear earphone, and that left/right distinction does notmatch the detected left/right distinction, the user may be warned tothat effect by the user interface with a display or sound.

Furthermore, it may also be configured such that when a given noddinggesture is conducted and a given goal is achieved, the user is informedto that effect with a display or sound (or voice).

In addition, it may also be configured such that an incorrect gesturemay be determined in the case where a given angle α discussed later isgreater than a predetermined angle. It may also be configured such thatthe user is instructed to retry the nodding gesture with a display orsound (or voice) in the case where the correct wearing of both earphonesis not detected after a given amount of time has elapsed since startingexecution of the application.

When the initialization process is completed (S12, Yes), the earphonewearing state monitoring behavior is started (S13). This involvesmonitoring the output from the acceleration sensors and/or geomagneticsensors, and determining which of the three wearing states discussedearlier corresponds to the current state.

Preset mobile device behavior corresponding to the current wearing stateis executed (S14).

Such mobile device behavior includes behavior such as launching orshutting down an application, switching the operational mode in anapplication, or switching the display in an application, for example.

After that, the wearing state of the left and right earphones ismonitored for any change, and in the case of change (S15, Yes), theprocess returns to step S14 and the mobile device behavior is made toreflect that change.

The process in FIG. 15 ends in response to the shutting down of theapplication executing the process, or in response to shutdowninstructions from the user, for example (S16, Yes).

FIG. 16 is a flowchart illustrating an exemplary modification of theprocess illustrated in FIG. 15. Like reference signs are given to stepswhich are similar to those illustrated in FIG. 15, and duplicatedescription thereof will be reduced or omitted.

The process in FIG. 16 includes an additional new step S17 along the Yesbranch of step S15 in FIG. 15. This step S17 is a step of determiningwhether or not a state has been confirmed when a change in the wearingstate has been determined. If a state is confirmed, the process proceedsto step S14. If a state is not confirmed, the process returns to theearphone initialization in the first step S11. In the case of anapplication that has detected that either the left or right earphone hasbeen removed, but has not specified (or is unable to specify) whetherthat earphone was the left or right earphone due to asymmetry in theleft and right sensor output, returning to the first initialization stepS11 makes it possible to re-execute monitoring of the earphone wearingstate starting from the initial state. Additionally, even forapplications that do specify whether the earphone was the left or rightearphone, the process in this exemplary modification still has value incases where left and right cannot be confirmed for some reason.

FIGS. 17(a) to 17(d) illustrate exemplary control of a mobile deviceaccording to the wearing state of left and right earphones. Thisexemplary control illustrates the control of a display interface andaudio output for a mobile device. This represents an example oftransitioning to pedestrian audio navigation that gives spoken streetdirections to a user in an application that displays a map on thedisplay screen of the mobile device.

FIG. 17(a) illustrates exemplary control when switching from a state inwhich neither earphone is worn (the third state) to a state in whichboth earphones are worn (the first state) while audio navigation is notbeing executed. In this example, a query message 61 asking “Do you wantto start audio navigation?” is displayed to the user, and audionavigation starts in response to the user's explicit instructions(pressing the “Yes” button).

FIG. 17(b) illustrates exemplary control when switching from the firststate in which both left and right earphones are worn, or from thesecond state in which one earphone is worn, to the state in which bothearphones are removed (the third state). In this example, a notificationmessage 62 stating “Audio navigation ended.” is presented to the user,and audio navigation ends.

FIG. 17(c) illustrates exemplary control when switching from the firststate in which both left and right earphones are worn to the secondstate in which one earphone is removed. In this example, a notificationmessage 63 stating “Audio navigation switched to monaural mode.” ispresented, and the audio output operational mode is switched from astereo mode to a monaural mode. The “monaural mode” referred to hereinis envisioned to emit the same audio output to the left and rightearphones, but may also emit audio output to only the earphone beingworn.

FIG. 17(d) illustrates exemplary control when returning from a state ofwearing one earphone (the second state) to the state of wearing bothearphones (the first state) while audio navigation is being executed. Inthis example, a notification message 64 stating “Audio navigationrestored to stereo mode.” is presented, and the audio output operationalmode is switched from monaural mode to stereo mode.

Note that although the example in FIG. 17(a) illustrates the case ofconfirming the user's intent and then launching an application orchanging the operational mode according to the user's explicitinstructions, it is not strictly necessary to confirm the user's intent.Conversely, it may also be configured such that a query message ispresented instead of a notification message in FIGS. 17(b) to 17(d), andcontrol is switched according to the user's explicit instructions.

FIGS. 18(a) and 18(b) illustrate exemplary displays that vary accordingto the wearing state of left and right earphones, which may beimplemented together with, or independently of, the exemplary controlfor audio navigation illustrated in FIGS. 17(a) to 17(d).

FIG. 18(a) is an exemplary display that presents a mark 71 indicatingthe current position on a map displayed on a display screen, while alsopresenting a user icon 72 indicating the user's travel direction at thatposition in a separate area. The user icon 72 is displayed in the caseof either the first or second state in which at least one of the leftand right earphones is being worn, and indicates that audio navigationis currently activated. In contrast, in the third state in which neitherof the left and right earphones are being worn, the user icon 72 is in ahidden display state, as in FIG. 18(b). This display state indicatesthat audio navigation is not currently activated. In this way, thewearing state of left and right earphones in this example is reflectedin whether or not a display element is displayed on a display screen,and also in whether or not a specific application is activated. However,an example of display control is not limited to this example.

Next, a method of using the above nodding gesture to more accuratelycompute the orientation F of the user's face by an orientation detectingunit of the exemplary embodiment will be described.

As illustrated in FIG. 19, the forward vector (Vf) of an earphone 10does not necessarily match the orientation F of the user's face while ina state where the earphone 10 is being worn on the head of the user 702.Thus, when the user wears the earphone 10, the angle differential θbetween the forward vector Vf and the orientation F of the face in thehorizontal plane is computed and stored on the basis of output from theacceleration sensor 12. Thereafter, while the earphone is being worn, itis possible to compute a correct orientation F of the user's face atthat time by correcting the direction of the forward vector Vf by theangle differential θ. Additionally, it is possible to compute theheading (such as north, south, east, or west) in which the user isfacing at that time by referring to output from the geomagnetic sensor11.

FIG. 20 once again illustrates a state in which the user 702 is wearingthe earphone 10, as well as a sensor coordinate system and usercoordinate system in such a state. The gravity vector G observed in therespective coordinate spaces may be expressed according to the followingEqs. 1 and 2.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 1} \right\rbrack \mspace{644mu}} & \; \\{{Gu} = \begin{pmatrix}{Gxu} \\{Gyu} \\{Gzu}\end{pmatrix}} & (1) \\{\left\lbrack {{Math}.\mspace{14mu} 2} \right\rbrack \mspace{644mu}} & \; \\{{Gs} = \begin{pmatrix}{Gxs} \\{Gys} \\{Gzs}\end{pmatrix}} & (2)\end{matrix}$

As illustrated in FIG. 21, axis transformation by rotation about the Zaxis of the earphone 10 is expressed in the following Eq. 3.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 3} \right\rbrack \mspace{644mu}} & \; \\{\begin{pmatrix}{Gxs} \\{Gys} \\{Gzs}\end{pmatrix} = {\begin{pmatrix}{Gxu} \\{Gyu} \\{Gzu}\end{pmatrix}\begin{pmatrix}{\cos \mspace{11mu} \varphi} & {{- \sin}\mspace{11mu} \varphi} & 0 \\{\sin \mspace{11mu} \varphi} & {\cos \mspace{11mu} \varphi} & 0 \\0 & 0 & 1\end{pmatrix}}} & (3)\end{matrix}$

Herein, the angle φ represents the tilt angle about the Z axis of the Ysaxis of the earphone 10 with respect to the Yu axis. In this case, theZs axis and the Zu axis are taken to approximately match. Gxs, Gys, andGzs are the axial components of the gravity vector G in the sensorcoordinate system, while Gxu, Gyu, and Gzu are the axial components ofthe gravity vector G in the user coordinate system.

Similarly, as illustrated in FIG. 22, axis transformation by rotation ofthe earphone 10 about the X axis is expressed in the following Eq. 4.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 4} \right\rbrack \mspace{644mu}} & \; \\{\begin{pmatrix}{Gxs} \\{Gys} \\{Gzs}\end{pmatrix} = {\begin{pmatrix}{Gxu} \\{Gyu} \\{Gzu}\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \mspace{11mu} \psi} & {{- \sin}\mspace{11mu} \psi} \\0 & {\sin \mspace{11mu} \psi} & {\cos \mspace{11mu} \psi}\end{pmatrix}}} & (4)\end{matrix}$

Herein, the angle ψ represents the tilt angle about the X axis of the Ysaxis of the earphone 10 with respect to the Yu axis. In this case, theXs axis and the Xu axis are taken to approximately match.

Also similarly, as illustrated in FIG. 23, axis transformation byrotation of the earphone 10 about the Y axis is expressed in thefollowing Eq. 5.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 5} \right\rbrack \mspace{644mu}} & \; \\{\begin{pmatrix}{Gxs} \\{Gys} \\{Gzs}\end{pmatrix} = {\begin{pmatrix}{Gxu} \\{Gyu} \\{Gzu}\end{pmatrix}\begin{pmatrix}{\cos \mspace{11mu} \theta} & 0 & {\sin \mspace{11mu} \theta} \\0 & 1 & 0 \\{{- \sin}\mspace{11mu} \theta} & 0 & {\cos \mspace{11mu} \theta}\end{pmatrix}}} & (5)\end{matrix}$

Herein, the angle θ represents the tilt angle about the Y axis of the Xsaxis of the earphone 10 with respect to the Xu axis. In this case, theYs axis and the Yu axis are taken to approximately match.

An axis transformation that takes into account the three angles φ, ψ,and θ from Eqs. 3, 4, and 5 is expressed in the following Eq. 6.

[Math.  6]                                            (6)$\begin{matrix}{\begin{pmatrix}{Gxs} \\{Gys} \\{Gzs}\end{pmatrix} = {\begin{pmatrix}{Gxu} \\{Gyu} \\{Gzu}\end{pmatrix}\begin{pmatrix}{\cos \; \varphi} & {{- \sin}\; \varphi} & 0 \\{\sin \; \varphi} & {\cos \; \varphi} & 0 \\0 & 0 & 1\end{pmatrix}\begin{pmatrix}1 & 0 & 0 \\0 & {\cos \; \psi} & {{- \sin}\; \psi} \\0 & {\sin \; \psi} & {\cos \; \psi}\end{pmatrix}\begin{pmatrix}{\cos \mspace{11mu} \theta} & 0 & {\sin \mspace{11mu} \theta} \\0 & 1 & 0 \\{{- \sin}\mspace{11mu} \theta} & 0 & {\cos \mspace{11mu} \theta}\end{pmatrix}}} \\{= \begin{pmatrix}\begin{matrix}{{{Gxu}\left( {{\cos \; {\varphi cos\theta}} - {\sin \; {\varphi cos}\; {\psi sin\theta}}} \right)} - {{Gyu}\left( {\sin \; {\varphi cos}\; \psi} \right)} +} \\{{Gzu}\left( {{\cos \; {\varphi sin\theta}} + {\sin \; {\varphi sin\psi cos\theta}}} \right)}\end{matrix} \\\begin{matrix}{{{Gxu}\left( {{\sin \; {\varphi cos\theta}} + {\cos \; {\varphi sin}\; {\psi sin\theta}}} \right)} + {{Gyu}\left( {\cos \; {\varphi cos}\; \psi} \right)} +} \\{{Gzu}\left( {{\cos \; {\varphi sin\theta}} + {\sin \; {\varphi sin\psi cos\theta}}} \right)}\end{matrix} \\{{- {{Gxu}\left( {\cos \; {\psi sin}\; \theta} \right)}} + {{Gyu}\left( {\sin \; \psi} \right)} + {{Gzu}\left( {\cos \; {\psi cos\theta}} \right)}}\end{pmatrix}}\end{matrix}$

At this point, if g is taken to be a constant expressing the absolutevalue of the gravitational force, the expression becomes like thefollowing Eq. 7.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 7} \right\rbrack \mspace{644mu}} & \; \\{{Gu} = {\begin{pmatrix}{Gxu} \\{Gyu} \\{Gzu}\end{pmatrix} = \begin{pmatrix}0 \\{- g} \\0\end{pmatrix}}} & (7)\end{matrix}$

Substituting this Gu into Eq. 6 yields the following Eq. 8.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 8} \right\rbrack \mspace{644mu}} & \; \\{\begin{pmatrix}{Gxs} \\{Gys} \\{Gzs}\end{pmatrix} = \begin{pmatrix}{g\; \sin \; {\varphi cos\psi}} \\{{- g}\; \cos \; {\varphi cos\psi}} \\{{- g}\; \sin \; \psi}\end{pmatrix}} & (8)\end{matrix}$

At this point, since g is a constant and the axial values Gxs, Gys, andGzs of Gs are ascertained from the output of the acceleration sensor, itis possible to compute the angles φ and ψ. However, the angle θ cannotbe computed.

Thus, as illustrated in FIG. 24, the user is made to execute a noddinggesture while in a state of wearing the earphone. With this gesture, thevertical plane containing the vector expressing the orientation F of theuser's face is determined. More specifically, when the user's headrotates in the vertical plane during the nodding gesture, the maximumrotational angle of the user's head with respect to the horizontal plane(the Xu-Yu plane), or in other words the maximum nodding angle α, iscomputed. The way to compute this angle α will be discussed later. Thegravity vector at the moment of this maximum nodding angle α is taken tobe a gravity vector G′. G′u may be expressed like the following Eq. 9.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 9} \right\rbrack \mspace{644mu}} & \; \\{{G^{\prime}u} = {\begin{pmatrix}{G^{\prime}{xu}} \\{G^{\prime}{yu}} \\{G^{\prime}{zu}}\end{pmatrix} = \begin{pmatrix}{g\mspace{11mu} \sin \mspace{11mu} \alpha} \\{{{- g}\mspace{11mu} \cos \mspace{11mu} \alpha}\mspace{11mu}} \\0\end{pmatrix}}} & (9)\end{matrix}$

Substituting this G′u (in other words, G′xu, G′yu, and G′zu) into theabove Eq. 6 yields the following Eq. 10.

$\begin{matrix}{\left\lbrack {{Math}.\mspace{14mu} 10} \right\rbrack \mspace{619mu}} & \; \\{\begin{pmatrix}{G^{\prime}{xs}} \\{G^{\prime}{ys}} \\{G^{\prime}{zs}}\end{pmatrix} = \begin{pmatrix}{{g\; \sin \; {\alpha \left( {{\cos \; {\varphi cos}\; \theta} - {\sin \; {\varphi cos\psi sin\theta}}} \right)}} + {g\; \cos \; {\alpha \left( {\sin \; {\varphi cos\psi}} \right)}}} \\{{{g\; \sin \; {\alpha \left( {{\sin \; {\varphi cos\theta}} + {\cos \; {\varphi sin}\; {\psi sin\theta}}} \right)}} - {g\; \cos \; {\alpha \left( {\cos \; {\varphi cos\psi}} \right)}}}\mspace{11mu}} \\{{{- g}\; \sin \; {\alpha \left( {\cos \; {\psi sin\theta}} \right)}} - {g\; \cos \; {\alpha \left( {\sin \; \psi} \right)}}}\end{pmatrix}} & (10)\end{matrix}$

The value of G's (in other words, G′xs, G′ys, and G′zs) is obtained fromthe output values of the acceleration sensor, and the values of theangles φ and ψ are known in the state before the nod. As a result, theangle θ can be computed. With this angle θ, it is possible to correcterror in the orientation of the user's face based on the forwarddirection of the earphone.

The way of computing the maximum nodding angle α will now be described,with reference to the graphs illustrating change in the gravity-inducedacceleration components Gys and Gxs during a nodding gesture illustratedin FIG. 11. As discussed earlier, both graphs are obtained by monitoringthe X axis and Y axis sensor output from the acceleration sensors over agiven interval at a given sampling period. As the graphs demonstrate,change in the extrema (maximum values) Gys(α) and Gxs(α) appears in thesensor output at the moment of the maximum nodding angle α. Thus, it ispossible to compute the angle α by monitoring for such extrema.

The maximum value is used because the precision of the computed angledecreases for non-maximum values due to noise in the acceleration valuefrom the inertial moment while the acceleration sensor is rotating dueto the nodding gesture. At the maximum angle, sensor motion momentarilystops, and noise is minimized.

A gyroscope may be used to further raise the detection precision for themaximum nodding angle α. Taking the rotational direction of thegyroscope during a nodding gesture to be about the a axis, the value ofthe gyroscope output Gyro-a varies like the sine waveform illustrated inFIG. 25 during the nodding gesture. At the moment when the noddinggesture by the user's head reaches the maximum angle, the gyroscoperotation stops, and its output becomes 0. For this reason, it becomespossible to more precisely compute the angle α by reading the outputfrom the acceleration sensor at the point when the gyroscope outputGyro-a becomes 0 (the zero-crossing point). However, use of a gyroscopeis not required.

In this way, even in the case where the earphone wearing angle withrespect to the user is offset from the expected wearing position in theXY plane and the YZ plane (the case where φ≠0 and ψ≠0), such tilt can bedetermined by the output from the acceleration sensor, as discussedabove. Consequently, the tilt θ in the XZ plane is similarly anduniquely determined by the nodding gesture, even from such an offsetstate.

Note that the foregoing description envisions the case where the twoearphones in a set of headphones are distinguished as left and right. Inthis case, the earphones are designed to generate convex sensor outputon both left and right while in a correct wearing state during a noddinggesture.

In contrast, in the case where the two earphones in a set of headphonesare not distinguished as left and right, both earphones may for examplegenerate convex sensor output when worn on the left and concave sensoroutput when worn on the right, without distinguishing between the leftand right acceleration sensors. Consequently, by checking the shape ofthe sensor output from the acceleration sensors, it is possible todetermine which side each earphone is being worn on.

Note that in this case, one of the waveforms of the left and rightsensor output will be convex during the nodding gesture as illustratedin FIG. 11, while the other will be concave. In this case, one of theleft and right waveforms may be reversed in sign when comparing thewaveforms (or alternatively, the absolute values of the waveforms may becompared).

In the case where the earphones are not distinguished as left and right,it must be confirmed which earphone is being worn on which ear (left orright) while being worn on the user's head, and stereo audio must becorrectly transmitted. The above is thus convenient since it is possibleto detect, on the basis of sensor output, whether each earphone is beingworn on the user's left or right ear. For example, in the case where twoearphones are not distinguished as left and right, and may be worn onarbitrary sides, it is determined which earphone is being worn on whichside after the earphones are put on. A mobile device may be configuredto subsequently conduct a switching control on the basis of the detectedresults, so as to send left or right audio output to the earphone on thecorresponding side.

The foregoing description envisions the case where the mobile device andthe headphones (earphones) are separate. However, a configuration inwhich the functionality of the mobile device is built into theheadphones is also conceivable. FIG. 26 illustrates an exemplaryconfiguration of such a mobile device 100 c with integrated headphones.This apparatus may also be interpreted to be headphones with built-inmobile device functionality.

An earphone speaker 421 a and mic 422 a are attached to the housing ofthe mobile device 100 c.

As illustrated in FIG. 26, in the case of stereo headphones, theconfiguration in FIG. 10 may be included in only one of the left andright earphones 10 bL and 10 bR (in this example, 10 bL). In this case,the earphone 10 bL is equipped with the wireless communication unit 108instead of the external connection terminal 107, and is wirelesslyconnected to the other earphone 10 bR. Alternatively, although notillustrated, the earphones may be connected to each other in a wiredmanner via the external connection terminal 107.

Note that not all of the components illustrated are required as themobile device 100 c. Furthermore, other components which are notillustrated, but which are provided in existing mobile devices, may alsobe included.

As described in the foregoing, an exemplary embodiment includes variousaspects like the following.

(1) A method of controlling a mobile device, including

a step of monitoring the wearing state of left and right earphones by auser, and

a step of controlling the behavior of a mobile device connected to theleft and right earphones according to the wearing state of the left andright earphones by the user.

(2) The method of controlling a mobile device according to (1), wherein

the wearing states of the left and right earphones include a first statein which both earphones are being worn, a second state in which only theleft or right earphone is being worn, and a third state in which boththe left and right earphones are removed.

(3) The method of controlling a mobile device according to (2), wherein

in the step of controlling the behavior of the mobile device, at leastone of launching or shutting down an application, switching theoperational mode in an application, and switching the display in anapplication is conducted, according to a change in the wearing state ofthe left and right earphones.

(4) The method of controlling a mobile device according to any of (1) to(3), wherein

in the step of monitoring the wearing state of the left and rightearphones, the wearing state of the left and right earphones is detectedon the basis of output from left and right acceleration sensorsrespectively provided in the left and right earphones, and/or outputfrom left and right geomagnetic sensors respectively provided in theleft and right earphones.

(5) The method of controlling a mobile device according to any of (1) to(3), further including

a step of causing the user to execute a nodding gesture, in which theuser looks directly ahead with respect to his or her body, rotates hisor her head forward from an upright state by a given angle or more, andthen returns his or her head to its original upright state, and

a step of checking whether each earphone is being worn, on the basis ofoutput during the nodding gesture from left and right accelerationsensors respectively provided in the left and right earphones.

(6) A mobile device, including

left and right earphones, and

a mobile device connectable to the left and right earphones,

wherein each of the left and right earphones includes an accelerationsensor and/or a geomagnetic sensor, and

the mobile device includes a controller that detects the wearing stateof the individual left and right earphones by the user on the basis ofoutput from the acceleration sensors and/or the geomagnetic sensors, andcontrols the behavior of the mobile device connected to the left andright earphones according to the wearing state of the left and rightearphones.

Although the foregoing describes a preferred embodiment, it is possibleto perform various alterations or modifications other than thosementioned above. In other words, it is to be understood as obvious bypersons skilled in the art that various modifications, combinations, andother embodiments may occur depending on design or other factors insofaras they are within the scope of the claims or their equivalents.

For example, although the gyroscope is described as not being requiredamong the multiple sensors on board an earphone, the geomagnetic sensoris also unnecessary if there is no need to compute the heading in whichthe user's face is facing.

A feature herein is the determining of whether an earphone is being wornon the correct side, depending on whether the output for a specific axisof the 3-axis acceleration sensor that varies during the nodding gestureexhibits convex variation or concave variation. However, this featuredoes not require actually computing the nodding angle α.

The present specification also encompasses a computer program forrealizing the functionality described in the foregoing exemplaryembodiments with a computer, as well as a recording medium storing sucha program in a computer-readable format. Potential examples of such arecording medium for supplying the program include magnetic storagemedia (such as a flexible disk, hard disk, or magnetic tape), opticaldiscs (such as an MO, PD, or other magneto-optical disc, a CD, or aDVD), and semiconductor storage, for example.

REFERENCE SIGNS LIST

-   -   10, 10L, 10R, 10 aL, 10 aR, 10 bL, 10 bR, 10 cL, 10 cR: earphone    -   10 a, 10 b: earphone (headphone)    -   11: 3-axis geomagnetic sensor    -   12: 3-axis acceleration sensor    -   13: gyroscope    -   15, 15 aL, 15 aR, 15 bL, 15 bR: housing    -   16 aL, 16 aR, 16 bL, 16 bR: sensor device    -   17 aL, 17 aR, 17 bL, 17 bR: ear canal plug    -   18, 18 i: cable    -   19, 19L, 19R: antenna    -   31: horizontal plane (plane)    -   33: plane    -   61: query message    -   62, 63, 64: notification message    -   71: mark    -   72: user icon    -   100, 100 a, 100 b, 100 c: mobile device    -   101: controller    -   102: communication circuit    -   103: antenna    -   104: display unit    -   105: operable unit    -   106: memory    -   107: external connection terminal    -   108: wireless communication unit    -   109: antenna    -   110: speaker    -   121: earphone terminal    -   122: mic    -   131: geomagnetic sensor    -   132: acceleration sensor    -   133: gyroscope    -   150: control line    -   160: data line    -   170: external apparatus connection unit    -   171: external storage apparatus    -   173: music data controller    -   174: imaging controller    -   175: camera unit    -   176: GPS controller    -   177: GPS antenna    -   401: external connection controller    -   411: external geomagnetic sensor    -   412: external acceleration sensor    -   413: external gyroscope    -   421: external speaker    -   421 a: speaker    -   422: external mic    -   422 a: mic    -   430: external wireless communication unit    -   431: external communication antenna    -   702: user

What is claimed is:
 1. A method of controlling a mobile device,comprising: monitoring a state of whether or not left and rightearphones are being worn by a user; and controlling a behavior of themobile device connected to the left and right earphones according to themonitored wearing state of the left and right earphones by the user.